Papers with speech processing
Self-supervised Representation Learning for Speech Processing (2022.naacl-tutorials)
Copied to clipboard
Hung-yi Lee, Abdelrahman Mohamed, Shinji Watanabe, Tara Sainath, Karen Livescu, Shang-Wen Li, Shu-wen Yang, Katrin Kirchhoff
| Challenge: | Self-supervised representation learning (SSL) uses proxy supervised learning tasks to obtain training data from unlabeled corpora. |
| Approach: | They propose to survey the latest SSL techniques, tools, datasets, and performance achievement in speech processing to scale up current machine learning technologies. |
| Outcome: | The proposed tutorial is highly relevant to the special theme of ACL about language diversity. |
ISA: An Intelligent Shopping Assistant (2020.aacl-demo)
Copied to clipboard
| Challenge: | In-store users only need to take a picture or scan the barcode of the product of interest, and then the user can talk to the assistant about the product. |
| Approach: | They present a mobile-based intelligent shopping assistant that is designed to improve shopping experience in physical stores. |
| Outcome: | The proposed system can improve shopping experience in physical stores by leveraging advanced techniques in computer vision, speech processing, and natural language processing. |
Towards efficient self-supervised representation learning in speech processing (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing models require several GPUs for days to pretrain, generating environmental concerns because of their high energy consumption. |
| Approach: | They propose an efficient self-supervised model that uses a single GPU during 24 to 48 hours of pretraining to address high computational costs. |
| Outcome: | The proposed model represents two orders of magnitude better than existing models. |
The taste of IPA: Towards open-vocabulary keyword spotting and forced alignment in any language (2024.naacl-long)
Copied to clipboard
| Challenge: | a recent study shows that multilingual speech processing systems can generalize to unseen languages without adaptation. |
| Approach: | They propose a phoneme-based phoneme embedding model that can be generalized to unseen languages by using a neural forced aligner. |
| Outcome: | The proposed model can generalize to unseen languages without adaptation. |
How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations (2025.naacl-short)
Copied to clipboard
| Challenge: | Recent advances in foundation models have sparked growing interest in expanding their text processing capabilities to speech. |
| Approach: | They analyze the model activations from semantically equivalent sentences across languages in the text and speech modalities and examine how text and spoken are represented in recent multimodal foundation models. |
| Outcome: | The proposed models exhibit cross-lingual differences, but are not explicitly trained for modality-agnostic representations. |
SeqXGPT: Sentence-Level AI-Generated Text Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for sentence-level AIGT detection are weak . large language models (LLMs) can generate human-like content . |
| Approach: | They propose a sentence-level AIGT detection challenge using LLMs as log probability lists . they propose 'check' GPT' method that uses log probability list features to detect AIGT . |
| Outcome: | The proposed method surpasses baseline methods in sentence- and document-level detection challenges. |
Praat++: Multimedia Annotation System for Speech and Vocalization (2026.acl-demo)
Copied to clipboard
| Challenge: | High-quality time-aligned annotation is fundamental to speech processing and animal vocalization research, yet precise boundary localization and consistent labeling remain challenging in collaborative settings. |
| Approach: | They propose a web-based multimedia annotation system for collaborative, video-informed, and AI-assisted timeline labeling of audio and video data. |
| Outcome: | The proposed system improves time-aligned labeling and accuracy in speech and animal vocalization annotations. |
Audiocite.net : A Large Spoken Read Dataset in French (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing self-supervised learning methods for speech processing have proved difficult to apply to French due to the scarcity of large speech datasets. |
| Approach: | They present a corpus of 6,682 hours of audiobooks from 130 readers . they describe the creation process and final statistics of the corpus . |
| Outcome: | The proposed model based on the audiocite.net corpus, which contains 6,682 hours of audiobooks, was able to perform in 14k version. |
Automatic Speech Interruption Detection: Analysis, Corpus, and System (2024.lrec-main)
Copied to clipboard
| Challenge: | Interruption detection is a new but challenging task in the field of speech processing. |
| Approach: | They propose to define automatic speech interruption detection and build a specialized corpus to analyze interrupted conversations. |
| Outcome: | The proposed system can detect interruptions in speech with promising results . it can be used to ensure speaking turns are respected during official political debates . |
Continual Contrastive Spoken Language Understanding (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in speech processing require extensive offline training . however, these models struggle to retain their previously acquired knowledge when learning new tasks continuously. |
| Approach: | They propose a method that relies on experience replay and contrastive learning to preserve the learned representations by pulling closer samples from the same class and pushing away the others. |
| Outcome: | The proposed method preserves the learned representations by pulling closer samples from the same class and pushing away the others. |
CHICA: A Developmental Corpus of Child-Caregiver’s Face-to-face vs. Video Call Conversations in Middle Childhood (2024.lrec-main)
Copied to clipboard
Dhia Elhak Goumri, Abhishek Agrawal, Mitja Nikolaus, Hong Duc Thang Vu, Kübra Bodur, Elias Emmar, Cassandre Armand, Chiara Mazzocconi, Shreejata Gupta, Laurent Prévot, Benoit Favre, Leonor Becerra-Bonache, Abdellah Fourtassi
| Challenge: | Existing studies of language-in-interaction focus on the two ends of the developmental spectrum, i.e., early childhood and adulthood, leaving a gap in our knowledge about how development unfolds, especially across middle childhood. |
| Approach: | They propose to use CHICA to analyze child-caregiver conversations at home . they use mobile, lightweight eye-tracking and head motion detection to optimize the naturalness of the recordings. |
| Outcome: | The proposed corpus of child-caregiver conversations at home was compared with a previous corpus based on a set of conversations between children aged 7, 9, and 11 years old. |
RepCodec: A Speech Representation Codec for Speech Tokenization (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have led to discrete speech tokenization, but this discretization can be costly and impedes performance. |
| Approach: | They propose a new speech representation codec for semantic speech tokenization that reconstructs speech representations from speech encoders like HuBERT or data2vec. |
| Outcome: | The proposed method outperforms the widely used k-means clustering approach in speech understanding and generation. |
Putting Natural in Natural Language Processing (2023.findings-acl)
Copied to clipboard
| Challenge: | human language is firstly spoken and only secondarily written. |
| Approach: | aaron carroll: human language is firstly spoken and only secondarily written . carroll says the field of NLP has overwhelmingly focused on processing written language . he says the focus is on a subset of human language which is convenient to work with . |
| Outcome: | the ACL 2023 theme track urges the community to check the reality of the progress in NLP . |
Fine-grained Artificial Neurons in Audio-transformers for Disentangling Neural Auditory Encoding (2023.findings-acl)
Copied to clipboard
Mengyue Zhou, Xu Liu, David Liu, Zihao Wu, Zhengliang Liu, Lin Zhao, Dajiang Zhu, Lei Guo, Junwei Han, Tianming Liu, Xintao Hu
| Challenge: | Existing studies treat each transformer encoding layer as a single artificial neuron . layer-level embeddings aggregate multiple types of contextual attention captured by multiple head modules . |
| Approach: | They propose to embed each transformer encoding layer as a single artificial neuron . they propose to couple those ANs with their biological-neuron counterparts in the human brain . |
| Outcome: | The proposed models can be used to link representations to brain activity, the authors say . their results show that the proposed models carry meaningful neurolinguistic information . |
Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformers (2023.emnlp-main)
Copied to clipboard
| Challenge: | 'context mixing' is a feature of Transformers that is used to build up representations of acoustic and linguistic structure in speech models. |
| Approach: | They propose to use a French spelling quirk to probe context mixing in speech models to find out how to translate spoken words into written equivalents. |
| Outcome: | The proposed model incorporates cues to identify correct transcription, whereas encoder-decoder models relegate task to decoder modules. |
Large Vocabulary Read Speech Corpora for Four Ethiopian Languages: Amharic, Tigrigna, Oromo and Wolaytta (2020.lrec-1)
Copied to clipboard
Solomon Teferra Abate, Martha Yifiru Tachbelie, Michael Melese, Hafte Abera, Tewodros Abebe, Wondwossen Mulugeta, Yaregal Assabie, Million Meshesha, Solomon Afnafu, Binyam Ephrem Seyoum
| Challenge: | Automatic Speech Recognition (ASR) is one of the most important technologies to support spoken communication in modern life. |
| Approach: | They have developed four large speech corpora for four Ethiopian languages . they have word error rates of 37.65%, 31.03%, 38.02%, 33.89% for each language . |
| Outcome: | The proposed corpora achieve word error rates of 37.65%, 31.03%, 38.02%, 33.89% for Amharic, Tigrigna, Oromo and Wolaytta. |
100,000 Podcasts: A Spoken English Document Corpus (2020.coling-main)
Copied to clipboard
Ann Clifton, Sravana Reddy, Yongze Yu, Aasish Pappu, Rezvaneh Rezapour, Hamed Bonab, Maria Eskevich, Gareth Jones, Jussi Karlgren, Ben Carterette, Rosie Jones
| Challenge: | Podcasts are a large and growing repository of spoken audio. |
| Approach: | They propose to use podcasts as a resource for speech processing and linguistics . they use a corpus of 100,000 podcasts to study the complexity of the domain . |
| Outcome: | The Spotify Podcast Dataset is the largest corpus of transcribed speech data . the dataset contains 60,000 hours of podcasts, with a range of genres and styles . |
Do self-supervised speech models develop human-like perception biases? (2022.acl-long)
Copied to clipboard
| Challenge: | Recent advances in speech recognition and representation learning show that self-supervised pretraining is an excellent way of improving performance while reducing the amount of labelled data needed for training. |
| Approach: | They compare the representational spaces of wav2vec, HuBERT and contrastive predictive coding (CPC) with the perceptual spaces of French-speaking and English-speaking human listeners. |
| Outcome: | The proposed models capture fine-grained perceptual phenomena while supervised models are better at capturing coarser, phone-level effects and effects of listeners’ native language on perception. |
Online Infix Probability Computation for Probabilistic Finite Automata (P19-1)
Copied to clipboard
| Challenge: | Probabilistic finite automata (PFAs) are statistical language models used in natural language processing. |
| Approach: | They develop an asymptotic algorithm to compute the infix probabilities of each prefix of a string from streaming data. |
| Outcome: | The proposed algorithm improves the infix probabilities of a weighted automata from streaming data. |
Context-aware Interactive Attention for Multi-modal Sentiment and Emotion Analysis (D19-1)
Copied to clipboard
| Challenge: | Multi-modal analysis is a field emerging in the fields of natural language processing, computer vision and speech processing . multimodal analysis uses a variety of information from multiple sources to build efficient systems . acoustic and visual information can provide better information for classification decisions . |
| Approach: | They propose a recurrent neural network based approach for multi-modal sentiment and emotion analysis . they employ a context-aware attention module to exploit the correspondence among neighboring utterances . |
| Outcome: | The proposed model learns inter-modal interaction among participating modalities through auto-encoder mechanism . it is compared with existing state-of-the-art models on five standard multi-modal affect analysis datasets . |
Experiments on Speech Synthesis for Teochew, Can Taiwanese Help ? (2024.lrec-main)
Copied to clipboard
| Challenge: | a recent uptick in interest in Teochew from heritage speakers of the diaspora has led to the development of a text-to-speech system. |
| Approach: | They develop a Teochew Text-to-Speech system to respond to the needs of the diaspora . they also use Taiwanese Hokkien to assess the contribution of available resources . |
| Outcome: | The proposed system is based on a Teochew Text-to-Speech system . the system is built on Taiwanese Hokkien, the closest language with a significant body of resources . |
The Distribution and Prosodic Realization of Verb Forms in German Infant-Directed Speech (L18-1)
Copied to clipboard
| Challenge: | Infant-directed speech is often seen as a predictor for infants' speech processing abilities, for instance speech segmentation or word learning. |
| Approach: | They examine the syntactic distribution, accentuation and prosodic phrasing of German verb forms and show that many verb forms are prime candidates for early segmentation. |
| Outcome: | The findings suggest that infants ought to be able to extract verbs as early as nouns, given appropriate stimulus materials. |
How Do Hyenas Deal with Human Speech? Speech Recognition and Translation with ConfHyena (2024.lrec-main)
Copied to clipboard
| Challenge: | Currently, attention-based models face computational hurdles in processing long sequences due to its quadratic complexity. |
| Approach: | They propose a conformer whose encoder self-attentions are replaced with Hyena for speech processing . they propose 'confhyena' model that reduces training time by 27% at minimal cost . |
| Outcome: | The proposed model reduces training time by 27% at the cost of minimal quality degradation. |
CTC-based Non-autoregressive Speech Translation (2023.acl-long)
Copied to clipboard
Chen Xu, Xiaoqian Liu, Xiaowen Liu, Qingxuan Sun, Yuhao Zhang, Murun Yang, Qianqian Dong, Tom Ko, Mingxuan Wang, Tong Xiao, Anxiang Ma, Jingbo Zhu
| Challenge: | End-to-end speech translation (E2E ST) and non-autoregressive (NAR) generation are promising in language and speech processing for their advantages of less error propagation and low latency. |
| Approach: | They develop a model that uses connectionist temporal classification to predict the source and target texts. |
| Outcome: | The proposed model achieves an average BLEU score of 29.5 with a speed-up of 5.67. |
Exploring Speaker-Related Information in Spoken Language Understanding for Better Speaker Diarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Current speaker diarization systems consider only acoustic information, resulting in performance degradation when encountering adverse acustic environment. |
| Approach: | They propose methods to extract speaker-related information from conversational semantics in multi-party meetings. |
| Outcome: | The proposed method improves on AISHELL-4 and AliMeeting datasets on speakers diarization and speaker-turn detection. |
Meta-Adapter for Self-Supervised Speech Models: A Solution to Low-Resource Speech Recognition Challenges (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing self-supervised learning models can learn latent representations from large amounts of unlabeled data, but they are expensive to fine-tune. |
| Approach: | They develop a meta-adapter to obtain meta-initialized parameters for self-supervised models . meta-Adapters show better generalization and extensibility than traditional pretraining methods . |
| Outcome: | Experiments on common voice and FLEURS datasets show Meta-Adapter performs better on low-resource languages . authors show it can be used on 12 low-source languages, but it requires huge computational resources . |
VocalNet: Speech LLMs with Multi-Token Prediction for Faster and High-Quality Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show VocalNet outperforms existing open-source speech LLMs despite limited training data. |
| Approach: | They propose a scalable and model-agnostic training framework and a novel multi-token prediction paradigm for speech generation. |
| Outcome: | The proposed model outperforms open-source speech LLMs while outperforming existing open-sourced models. |
myMediCon: End-to-End Burmese Automatic Speech Recognition for Medical Conversations (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing medical conversation speech corpora for Burmese are limited, despite advances in ASR. |
| Approach: | They propose to use a manually curated medical conversation speech corpus for Burmese to examine the performance of ASR models. |
| Outcome: | The proposed model outperforms the Transformer model and the Recurrent Neural Network (RNN) models. |
Project MOSLA: Recording Every Moment of Second Language Acquisition (2024.lrec-main)
Copied to clipboard
| Challenge: | Second language acquisition (SLA) is a complex and dynamic process. |
| Approach: | They created a longitudinal, multimodal, multilingual, and controlled dataset by inviting participants to learn one of three target languages from scratch over a span of two years, exclusively through online instruction. |
| Outcome: | The proposed dataset sheds light on the complex and dynamic nature of the acquisition of a second language and its implications for proficiency assessment, language and speech processing, and multimodal learning analytics. |
Discriminating Form and Meaning in Multilingual Models with Minimal-Pair ABX Tasks (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that multilingual models encode languagespecific information and language-agnostic features, but the nature and interaction of these representations is not fully understood. |
| Approach: | They propose a set of training-free ABX-style discrimination tasks to evaluate how multilingual language models represent language identity (form) and semantic content (meaning). |
| Outcome: | The proposed tasks show that language discrimination declines over training and strengthens over time and stabilizes in deeper layers. |
Casablanca: Data and Models for Multidialectal Arabic Speech Recognition (2024.emnlp-main)
Copied to clipboard
Bashar Talafha, Karima Kadaoui, Samar Magdy, Mariem Habiboullah, Chafei Chafei, Ahmed El-Shangiti, Hiba Zayed, Mohamedou Tourad, Rahaf Alhamouri, Rwaa Assi, Aisha Alraeesi, Hour Mohamed, Fakhraddin Alwajih, Abdelrahman Mohamed, Abdellah El Mekki, El Moatez Billah Nagoudi, Benelhadj Saadia, Hamzah Alsayadi, Walid Al-Dhabyani, Sara Shatnawi, Yasir Ech-chammakhy, Amal Makouar, Yousra Berrachedi, Mustafa Jarrar, Shady Shehata, Ismail Berrada, Muhammad Abdul-Mageed
| Challenge: | despite recent advances in speech processing, the majority of world languages and dialects remain uncovered. |
| Approach: | They propose to collect and transcribe a new Arabic dataset for eight dialects . they also develop strong baselines exploiting the new dataset . |
| Outcome: | The proposed dataset covers eight Arabic dialects, including Algerian, Egyptian, Emirati, Jordanian, Mauritanian, Moroccan, Palestinian, and Yemeni. |
Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Speech Large Language Models (SpeechLLMs) have emerged as dominant speech processing approaches. |
| Approach: | They compare self-supervised learning-based discrete and continuous features . they compare performance across six spoken language understanding-related tasks . |
| Outcome: | The proposed models outperform discrete tokens and continuous features in six spoken language understanding-related tasks. |
WenetSpeech-Wu: Datasets, Benchmarks, and Models for a Unified Chinese Wu Dialect Speech Processing Ecosystem (2026.findings-acl)
Copied to clipboard
Chengyou Wang, Mingchen Shao, Jingbin Hu, Zeyu Zhu, Hongfei Xue, Bingshen Mu, Xin Xu, Xingyi Duan, Binbin Zhang, Zhu Pengcheng, Chuang Ding, Xiaojun Zhang, Hui Bu, Lei Xie
| Challenge: | despite its linguistic significance, the Wu dialect of Chinese has long been hindered by the lack of large-scale speech data, standardized evaluation benchmarks, and publicly available models. |
| Approach: | They propose to use WenetSpeech-Wu as a large-scale, multi-dimensionally annotated open-source speech corpus for the Wu dialect of Chinese. |
| Outcome: | The proposed dataset includes 8,000 hours of speech data and strong open-source models . the proposed dataset is competitive and empirically validated . |
Unraveling Spontaneous Speech Dimensions for Cross-Corpus ASR System Evaluation for French (2024.lrec-main)
Copied to clipboard
| Challenge: | 'spontaneous speech' is a catch-all term used for situations like speaking with a friend, being interviewed on radio/TV or giving a lecture. |
| Approach: | They propose to use four dimensions to describe spontaneous speech variation in automatic speech recognition systems. |
| Outcome: | The proposed system can be used to predict the WER of speech recognition systems on face-to-face interactions. |